{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Criteo Uplift Modeling Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset is released along with the paper: “A Large Scale Benchmark for Uplift Modeling” Eustache Diemert, Artem Betlei, Christophe Renaudin; (Criteo AI Lab), Massih-Reza Amini (LIG, Grenoble INP) This work was published in: AdKDD 2018 Workshop, in conjunction with KDD 2018.\n",
    "\n",
    "This dataset is constructed by assembling data resulting from several incrementality tests, a particular randomized trial procedure where a random part of the population is prevented from being targeted by advertising."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a detailed description of the fields original dataset:\n",
    "\n",
    "- **f0, f1, f2, f3, f4, f5, f6, f7, f8, f9, f10, f11**: feature values (dense, float)\n",
    "\n",
    "- **treatment**: treatment group. Flag if a company participates in the RTB auction for a particular user (binary: 1 = treated, 0 = control)\n",
    "\n",
    "- **exposure**: treatment effect, whether the user has been effectively exposed. Flag if a company wins in the RTB auction for the user (binary)\n",
    "\n",
    "- **conversion**: whether a conversion occured for this user (binary, label)\n",
    "\n",
    "- **visit**: whether a visit occured for this user (binary, label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "# install uplift library scikit-uplift and other libraries \n",
    "!{sys.executable} -m pip install scikit-uplift dill lightgbm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklift.datasets import fetch_criteo\n",
    "import pandas as pd \n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklift.models import TwoModels\n",
    "import lightgbm as lgb\n",
    "from sklift.metrics import qini_auc_score\n",
    "from sklift.viz import plot_qini_curve\n",
    "\n",
    "seed=31"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📝 Load data\n",
    "\n",
    "Dataset can be loaded from `sklift.datasets` module using `fetch_criteo` function. There are few function parameters:\n",
    "- target_col – selects which column ('visit', 'conversion' or 'all') from dataset will be target\n",
    "- treatment_col – selects which column ('treatment', 'exposure' or 'all') from dataset will be treatment.\n",
    "- percent10 – whether to load only 10 percent of the data.\n",
    "\n",
    "Let's load the dataset with default parameters (target = 'visit', treatment = 'treatment'). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# returns sklearn Bunch object\n",
    "# with data, target, treatment keys\n",
    "# data features (pd.DataFrame), target (pd.Series), treatment (pd.Series) values \n",
    "\n",
    "dataset = fetch_criteo()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset type: <class 'sklearn.utils.Bunch'>\n",
      "\n",
      "Dataset features shape: (13979592, 12)\n",
      "Dataset target shape: (13979592,)\n",
      "Dataset treatment shape: (13979592,)\n"
     ]
    }
   ],
   "source": [
    "print(f\"Dataset type: {type(dataset)}\\n\")\n",
    "print(f\"Dataset features shape: {dataset.data.shape}\")\n",
    "print(f\"Dataset target shape: {dataset.target.shape}\")\n",
    "print(f\"Dataset treatment shape: {dataset.treatment.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📝 EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the data features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 13979592 entries, 0 to 13979591\n",
      "Data columns (total 12 columns):\n",
      " #   Column  Dtype  \n",
      "---  ------  -----  \n",
      " 0   f0      float64\n",
      " 1   f1      float64\n",
      " 2   f2      float64\n",
      " 3   f3      float64\n",
      " 4   f4      float64\n",
      " 5   f5      float64\n",
      " 6   f6      float64\n",
      " 7   f7      float64\n",
      " 8   f8      float64\n",
      " 9   f9      float64\n",
      " 10  f10     float64\n",
      " 11  f11     float64\n",
      "dtypes: float64(12)\n",
      "memory usage: 1.2 GB\n"
     ]
    }
   ],
   "source": [
    "dataset.data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>f0</th>\n",
       "      <th>f1</th>\n",
       "      <th>f2</th>\n",
       "      <th>f3</th>\n",
       "      <th>f4</th>\n",
       "      <th>f5</th>\n",
       "      <th>f6</th>\n",
       "      <th>f7</th>\n",
       "      <th>f8</th>\n",
       "      <th>f9</th>\n",
       "      <th>f10</th>\n",
       "      <th>f11</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>8.976429</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.955396</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>9.002689</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.955396</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>8.964775</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.955396</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>9.002801</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.955396</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>9.037999</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.955396</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13979587</th>\n",
       "      <td>26.297764</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>9.006250</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>-3.282109</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.839578</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13979588</th>\n",
       "      <td>12.642207</td>\n",
       "      <td>10.679513</td>\n",
       "      <td>8.214383</td>\n",
       "      <td>-1.700105</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>3.013064</td>\n",
       "      <td>-13.955150</td>\n",
       "      <td>6.269026</td>\n",
       "      <td>3.971858</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13979589</th>\n",
       "      <td>12.976557</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>8.381868</td>\n",
       "      <td>0.842442</td>\n",
       "      <td>11.029584</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>-8.281971</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.779212</td>\n",
       "      <td>23.570168</td>\n",
       "      <td>6.169187</td>\n",
       "      <td>-0.168679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13979590</th>\n",
       "      <td>24.805064</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>8.214383</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>-1.288207</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.971858</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13979591</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>8.214383</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>3.013064</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>9.332563</td>\n",
       "      <td>3.971858</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 f0         f1        f2        f3         f4        f5  \\\n",
       "0         12.616365  10.059654  8.976429  4.679882  10.280525  4.115453   \n",
       "1         12.616365  10.059654  9.002689  4.679882  10.280525  4.115453   \n",
       "2         12.616365  10.059654  8.964775  4.679882  10.280525  4.115453   \n",
       "3         12.616365  10.059654  9.002801  4.679882  10.280525  4.115453   \n",
       "4         12.616365  10.059654  9.037999  4.679882  10.280525  4.115453   \n",
       "13979587  26.297764  10.059654  9.006250  4.679882  10.280525  4.115453   \n",
       "13979588  12.642207  10.679513  8.214383 -1.700105  10.280525  3.013064   \n",
       "13979589  12.976557  10.059654  8.381868  0.842442  11.029584  4.115453   \n",
       "13979590  24.805064  10.059654  8.214383  4.679882  10.280525  4.115453   \n",
       "13979591  12.616365  10.059654  8.214383  4.679882  10.280525  3.013064   \n",
       "\n",
       "                 f6        f7        f8         f9       f10       f11  \n",
       "0          0.294443  4.833815  3.955396  13.190056  5.300375 -0.168679  \n",
       "1          0.294443  4.833815  3.955396  13.190056  5.300375 -0.168679  \n",
       "2          0.294443  4.833815  3.955396  13.190056  5.300375 -0.168679  \n",
       "3          0.294443  4.833815  3.955396  13.190056  5.300375 -0.168679  \n",
       "4          0.294443  4.833815  3.955396  13.190056  5.300375 -0.168679  \n",
       "13979587  -3.282109  4.833815  3.839578  13.190056  5.300375 -0.168679  \n",
       "13979588 -13.955150  6.269026  3.971858  13.190056  5.300375 -0.168679  \n",
       "13979589  -8.281971  4.833815  3.779212  23.570168  6.169187 -0.168679  \n",
       "13979590  -1.288207  4.833815  3.971858  13.190056  5.300375 -0.168679  \n",
       "13979591   0.294443  9.332563  3.971858  13.190056  5.300375 -0.168679  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.data.head().append(dataset.data.tail())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>f0</th>\n",
       "      <th>f1</th>\n",
       "      <th>f2</th>\n",
       "      <th>f3</th>\n",
       "      <th>f4</th>\n",
       "      <th>f5</th>\n",
       "      <th>f6</th>\n",
       "      <th>f7</th>\n",
       "      <th>f8</th>\n",
       "      <th>f9</th>\n",
       "      <th>f10</th>\n",
       "      <th>f11</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "      <td>1.397959e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.962030e+01</td>\n",
       "      <td>1.006998e+01</td>\n",
       "      <td>8.446582e+00</td>\n",
       "      <td>4.178923e+00</td>\n",
       "      <td>1.033884e+01</td>\n",
       "      <td>4.028513e+00</td>\n",
       "      <td>-4.155356e+00</td>\n",
       "      <td>5.101765e+00</td>\n",
       "      <td>3.933581e+00</td>\n",
       "      <td>1.602764e+01</td>\n",
       "      <td>5.333396e+00</td>\n",
       "      <td>-1.709672e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>5.377464e+00</td>\n",
       "      <td>1.047557e-01</td>\n",
       "      <td>2.993161e-01</td>\n",
       "      <td>1.336645e+00</td>\n",
       "      <td>3.433081e-01</td>\n",
       "      <td>4.310974e-01</td>\n",
       "      <td>4.577914e+00</td>\n",
       "      <td>1.205248e+00</td>\n",
       "      <td>5.665958e-02</td>\n",
       "      <td>7.018975e+00</td>\n",
       "      <td>1.682288e-01</td>\n",
       "      <td>2.283277e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.261636e+01</td>\n",
       "      <td>1.005965e+01</td>\n",
       "      <td>8.214383e+00</td>\n",
       "      <td>-8.398387e+00</td>\n",
       "      <td>1.028053e+01</td>\n",
       "      <td>-9.011892e+00</td>\n",
       "      <td>-3.142978e+01</td>\n",
       "      <td>4.833815e+00</td>\n",
       "      <td>3.635107e+00</td>\n",
       "      <td>1.319006e+01</td>\n",
       "      <td>5.300375e+00</td>\n",
       "      <td>-1.383941e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.261636e+01</td>\n",
       "      <td>1.005965e+01</td>\n",
       "      <td>8.214383e+00</td>\n",
       "      <td>4.679882e+00</td>\n",
       "      <td>1.028053e+01</td>\n",
       "      <td>4.115453e+00</td>\n",
       "      <td>-6.699321e+00</td>\n",
       "      <td>4.833815e+00</td>\n",
       "      <td>3.910792e+00</td>\n",
       "      <td>1.319006e+01</td>\n",
       "      <td>5.300375e+00</td>\n",
       "      <td>-1.686792e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2.192341e+01</td>\n",
       "      <td>1.005965e+01</td>\n",
       "      <td>8.214383e+00</td>\n",
       "      <td>4.679882e+00</td>\n",
       "      <td>1.028053e+01</td>\n",
       "      <td>4.115453e+00</td>\n",
       "      <td>-2.411115e+00</td>\n",
       "      <td>4.833815e+00</td>\n",
       "      <td>3.971858e+00</td>\n",
       "      <td>1.319006e+01</td>\n",
       "      <td>5.300375e+00</td>\n",
       "      <td>-1.686792e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.443646e+01</td>\n",
       "      <td>1.005965e+01</td>\n",
       "      <td>8.723335e+00</td>\n",
       "      <td>4.679882e+00</td>\n",
       "      <td>1.028053e+01</td>\n",
       "      <td>4.115453e+00</td>\n",
       "      <td>2.944427e-01</td>\n",
       "      <td>4.833815e+00</td>\n",
       "      <td>3.971858e+00</td>\n",
       "      <td>1.319006e+01</td>\n",
       "      <td>5.300375e+00</td>\n",
       "      <td>-1.686792e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2.674526e+01</td>\n",
       "      <td>1.634419e+01</td>\n",
       "      <td>9.051962e+00</td>\n",
       "      <td>4.679882e+00</td>\n",
       "      <td>2.112351e+01</td>\n",
       "      <td>4.115453e+00</td>\n",
       "      <td>2.944427e-01</td>\n",
       "      <td>1.199840e+01</td>\n",
       "      <td>3.971858e+00</td>\n",
       "      <td>7.529502e+01</td>\n",
       "      <td>6.473917e+00</td>\n",
       "      <td>-1.686792e-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 f0            f1            f2            f3            f4  \\\n",
       "count  1.397959e+07  1.397959e+07  1.397959e+07  1.397959e+07  1.397959e+07   \n",
       "mean   1.962030e+01  1.006998e+01  8.446582e+00  4.178923e+00  1.033884e+01   \n",
       "std    5.377464e+00  1.047557e-01  2.993161e-01  1.336645e+00  3.433081e-01   \n",
       "min    1.261636e+01  1.005965e+01  8.214383e+00 -8.398387e+00  1.028053e+01   \n",
       "25%    1.261636e+01  1.005965e+01  8.214383e+00  4.679882e+00  1.028053e+01   \n",
       "50%    2.192341e+01  1.005965e+01  8.214383e+00  4.679882e+00  1.028053e+01   \n",
       "75%    2.443646e+01  1.005965e+01  8.723335e+00  4.679882e+00  1.028053e+01   \n",
       "max    2.674526e+01  1.634419e+01  9.051962e+00  4.679882e+00  2.112351e+01   \n",
       "\n",
       "                 f5            f6            f7            f8            f9  \\\n",
       "count  1.397959e+07  1.397959e+07  1.397959e+07  1.397959e+07  1.397959e+07   \n",
       "mean   4.028513e+00 -4.155356e+00  5.101765e+00  3.933581e+00  1.602764e+01   \n",
       "std    4.310974e-01  4.577914e+00  1.205248e+00  5.665958e-02  7.018975e+00   \n",
       "min   -9.011892e+00 -3.142978e+01  4.833815e+00  3.635107e+00  1.319006e+01   \n",
       "25%    4.115453e+00 -6.699321e+00  4.833815e+00  3.910792e+00  1.319006e+01   \n",
       "50%    4.115453e+00 -2.411115e+00  4.833815e+00  3.971858e+00  1.319006e+01   \n",
       "75%    4.115453e+00  2.944427e-01  4.833815e+00  3.971858e+00  1.319006e+01   \n",
       "max    4.115453e+00  2.944427e-01  1.199840e+01  3.971858e+00  7.529502e+01   \n",
       "\n",
       "                f10           f11  \n",
       "count  1.397959e+07  1.397959e+07  \n",
       "mean   5.333396e+00 -1.709672e-01  \n",
       "std    1.682288e-01  2.283277e-02  \n",
       "min    5.300375e+00 -1.383941e+00  \n",
       "25%    5.300375e+00 -1.686792e-01  \n",
       "50%    5.300375e+00 -1.686792e-01  \n",
       "75%    5.300375e+00 -1.686792e-01  \n",
       "max    6.473917e+00 -1.686792e-01  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number NA: 0\n"
     ]
    }
   ],
   "source": [
    "print('Number NA:', dataset.data.isna().sum().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some notes:\n",
    "\n",
    "- There are 12 columns in the dataset\n",
    "- All feature values are float\n",
    "- There are no missing values in data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Target and treatment "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also take a look at target and treatment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fbedb888650>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAERCAYAAAB2CKBkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAQkUlEQVR4nO3df6yeZX3H8feHFmRO8FePm7ZAG1edjcMxz9DoMvHXUkzWLo6xNjKdVuucYIzODDODDrNkiotRB2rjEDEThmy6ulRrpjicivQwEWgZrilOTnByBMRfG1D97o/nOfp4+pz2sZz7PD1c71dyp/d1X9dz3982p8/n3L9TVUiS2nXUuAuQJI2XQSBJjTMIJKlxBoEkNc4gkKTGGQSS1LglGQRJLk5yR5KbRhj7ziTX96evJfnOYtQoSUtFluJ9BEl+G/g+cGlVPfnn+Nw5wClV9bLOipOkJWZJ7hFU1dXAXYPLkjw+yaeSXJfk80l+dchHNwOXLUqRkrRELB93AQtoG/AnVfVfSZ4GXAQ8Z7YzyUnAGuCzY6pPko5ID4ogSPIw4BnAR5PMLn7InGGbgCur6keLWZskHekeFEFA7xDXd6rq1w8yZhPw6kWqR5KWjCV5jmCuqvoucGuSPwBIz1Nm+5M8EXgk8KUxlShJR6wlGQRJLqP3pf7EJNNJtgAvArYk+SqwG9g48JHNwOW1FC+RkqSOLcnLRyVJC2dJ7hFIkhbOkjtZvGLFilq9evW4y5CkJeW66677dlVNDOtbckGwevVqpqamxl2GJC0pSf57vj4PDUlS4wwCSWqcQSBJjTMIJKlxnQXBod4ZkORFSW7oT18cvBNYkrR4utwjuARYf5D+W4FnVdXJwFvpPT1UkrTIOrt8tKquTrL6IP1fHGheA6zqqhZJ0vyOlHMEW4BPzteZZGuSqSRTMzMzi1iWJD34jT0IkjybXhD8+XxjqmpbVU1W1eTExNAb4yRJh2msdxYnORn4AHB6Vd05zlqkI8E3zv+1cZegI9CJ593Y6frHtkeQ5ETgn4A/qqqvjasOSWpdZ3sE/XcGnAasSDINvBk4GqCq3gecBzwauKj/esn9VTXZVT2SpOG6vGpo8yH6Xw68vKvtS5JGM/aTxZKk8TIIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjessCJJcnOSOJDfN058k706yN8kNSX6jq1okSfPrco/gEmD9QfpPB9b2p63AezusRZI0j86CoKquBu46yJCNwKXVcw3wiCSP7aoeSdJw4zxHsBK4baA93V8mSVpE4wyCDFlWQwcmW5NMJZmamZnpuCxJass4g2AaOGGgvQq4fdjAqtpWVZNVNTkxMbEoxUlSK8YZBNuBF/evHno6cE9VfXOM9UhSk5Z3teIklwGnASuSTANvBo4GqKr3ATuAFwB7gR8CL+2qFknS/DoLgqrafIj+Al7d1fYlSaPxzmJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmN6zQIkqxPckuSvUnOHdJ/YpKrknwlyQ1JXtBlPZKkA3UWBEmWARcCpwPrgM1J1s0Z9ibgiqo6BdgEXNRVPZKk4brcIzgV2FtV+6rqPuByYOOcMQUc359/OHB7h/VIkoboMghWArcNtKf7ywa9BTgryTSwAzhn2IqSbE0ylWRqZmami1olqVldBkGGLKs57c3AJVW1CngB8OEkB9RUVduqarKqJicmJjooVZLa1WUQTAMnDLRXceChny3AFQBV9SXgWGBFhzVJkuboMgh2AWuTrElyDL2TwdvnjPkG8FyAJE+iFwQe+5GkRdRZEFTVfuBsYCdwM72rg3YnOT/Jhv6w1wOvSPJV4DLgj6tq7uEjSVKHlne58qraQe8k8OCy8wbm9wDP7LIGSdLBeWexJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWrcSEGQ5DOjLJMkLT0HfR9BkmOBhwIrkjySn76H+HjgcR3XJklaBId6Mc0rgdfS+9K/jp8GwXeBCzusS5K0SA4aBFX1LuBdSc6pqvcsUk2SpEU00qsqq+o9SZ4BrB78TFVd2lFdkqRFMlIQJPkw8HjgeuBH/cUFGASStMSN+vL6SWBdVVWXxUiSFt+o9xHcBPxyl4VIksZj1D2CFcCeJNcC984urKoNnVQlSVo0owbBW7osQpI0PqNeNfRvXRciSRqPUa8a+h69q4QAjgGOBn5QVcd3VZgkaXGMdLK4qo6rquP707HA7wN/e6jPJVmf5JYke5OcO8+YM5PsSbI7yUd+vvIlSQ/UqOcIfkZVfXy+L/ZZSZbRewzF84FpYFeS7VW1Z2DMWuCNwDOr6u4kjzmceiRJh2/UQ0MvHGgeRe++gkPdU3AqsLeq9vXXcTmwEdgzMOYVwIVVdTdAVd0xYt2SpAUy6h7B7w7M7we+Tu9L/WBWArcNtKeBp80Z8wSAJF8AlgFvqapPzV1Rkq3AVoATTzxxxJIlSaMY9aqhlx7GujNk2dy9iOXAWuA0YBXw+SRPrqrvzNn+NmAbwOTkpHc3S9ICGvXFNKuSfCzJHUm+leQfk6w6xMemgRMG2quA24eM+eequr+qbgVuoRcMkqRFMuojJj4IbKf3XoKVwCf6yw5mF7A2yZokxwCb+usY9HHg2QBJVtA7VLRvxJokSQtg1CCYqKoPVtX+/nQJMHGwD1TVfuBsYCdwM3BFVe1Ocn6S2UdT7ATuTLIHuAp4Q1XdeVh/E0nSYRn1ZPG3k5wFXNZvbwYO+YVdVTuAHXOWnTcwX8Dr+pMkaQxG3SN4GXAm8D/AN4EzgMM5gSxJOsKMukfwVuAls9f7J3kU8A56ASFJWsJG3SM4eTYEAKrqLuCUbkqSJC2mUYPgqCSPnG309wgO6/EUkqQjy6hf5n8DfDHJlfRuCjsT+KvOqpIkLZpR7yy+NMkU8Bx6dwy/cPDhcZKkpWvkwzv9L36//CXpQWbUcwSSpAcpg0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuM6DYIk65PckmRvknMPMu6MJJVksst6JEkH6iwIkiwDLgROB9YBm5OsGzLuOOA1wJe7qkWSNL8u9whOBfZW1b6qug+4HNg4ZNxbgbcD/9dhLZKkeXQZBCuB2wba0/1lP5HkFOCEqvqXg60oydYkU0mmZmZmFr5SSWpYl0GQIcvqJ53JUcA7gdcfakVVta2qJqtqcmJiYgFLlCR1GQTTwAkD7VXA7QPt44AnA59L8nXg6cB2TxhL0uLqMgh2AWuTrElyDLAJ2D7bWVX3VNWKqlpdVauBa4ANVTXVYU2SpDk6C4Kq2g+cDewEbgauqKrdSc5PsqGr7UqSfj7Lu1x5Ve0AdsxZdt48Y0/rshZJ0nDeWSxJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxnQZBkvVJbkmyN8m5Q/pfl2RPkhuSfCbJSV3WI0k60PKuVpxkGXAh8HxgGtiVZHtV7RkY9hVgsqp+mORVwNuBP+yqpllPfcOlXW9CS9B1F7x43CVIY9HlHsGpwN6q2ldV9wGXAxsHB1TVVVX1w37zGmBVh/VIkoboMghWArcNtKf7y+azBfjksI4kW5NMJZmamZlZwBIlSV0GQYYsq6EDk7OASeCCYf1Vta2qJqtqcmJiYgFLlCR1do6A3h7ACQPtVcDtcwcleR7wF8CzqureDuuRJA3R5R7BLmBtkjVJjgE2AdsHByQ5BXg/sKGq7uiwFknSPDoLgqraD5wN7ARuBq6oqt1Jzk+yoT/sAuBhwEeTXJ9k+zyrkyR1pMtDQ1TVDmDHnGXnDcw/r8vtS5IOzTuLJalxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNa7TIEiyPsktSfYmOXdI/0OS/EO//8tJVndZjyTpQJ0FQZJlwIXA6cA6YHOSdXOGbQHurqpfAd4JvK2reiRJw3W5R3AqsLeq9lXVfcDlwMY5YzYCH+rPXwk8N0k6rEmSNMfyDte9ErhtoD0NPG2+MVW1P8k9wKOBbw8OSrIV2Npvfj/JLZ1U3KYVzPn3blXe8ZJxl6Cf5c/mrDcvyO/HJ83X0WUQDKu8DmMMVbUN2LYQRelnJZmqqslx1yHN5c/m4uny0NA0cMJAexVw+3xjkiwHHg7c1WFNkqQ5ugyCXcDaJGuSHANsArbPGbMdmN0fPwP4bFUdsEcgSepOZ4eG+sf8zwZ2AsuAi6tqd5Lzgamq2g78HfDhJHvp7Qls6qoezctDbjpS+bO5SOIv4JLUNu8slqTGGQSS1DiDoFGHevyHNC5JLk5yR5Kbxl1LKwyCBo34+A9pXC4B1o+7iJYYBG0a5fEf0lhU1dV4P9GiMgjaNOzxHyvHVIukMTMI2jTSoz0ktcEgaNMoj/+Q1AiDoE2jPP5DUiMMggZV1X5g9vEfNwNXVNXu8VYl9SS5DPgS8MQk00m2jLumBzsfMSFJjXOPQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBmpHkEUn+dAHX99okD12o9Y17O2qXQaCWPAI4IAj6T2M9HK8FFuMLerG2o0YZBGrJXwOPT3J9kl1JrkryEeBGgCRnJbm23//+2YBI8t4kU0l2J/nL/rLXAI8DrkpyVX/Z95O8Lcl1Sf41yalJPpdkX5IN/THLklzQ3/4NSV7ZX35af+yVSf4zyd+n54DtSAuuqpycmpiA1cBN/fnTgB8Aa/rtJwGfAI7uty8CXtyff1T/z2XA54CT++2vAysG1l/A6f35jwGfBo4GngJc31++FXhTf/4hwBSwpl/PPfSe+3QUvTtrf2vYdpycFnpavpChIi0x11bVrf355wJPBXYlAfgF4I5+35lJtgLLgcfSe5nPDUPWdx/wqf78jcC9VXV/khvphRDA7wAnJzmj3344sLb/2WurahogyfX9z/z7A/9rSgdnEKhlPxiYD/Chqnrj4IAka4A/A36zqu5Ocglw7Dzru7+qZp/Z8mPgXoCq+nGS2f9rAc6pqp1ztnPa7Pi+H+H/Ty0SzxGoJd8Djpun7zPAGUkeA5DkUUlOAo6nFxj3JPkleq/3HGV989kJvCrJ0f3tPCHJLz6AuqUHzN841IyqujPJF/ovRf9f4FsDfXuSvAn4dJKjgPuBV1fVNUm+AuwG9gFfGFjlNuCTSb5ZVc8esYwP0Dvk8x/pHYOaAX7vEJ85nO1II/Ppo5LUOA8NSVLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUuP8HZDwA1PHgLrwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(x=dataset.treatment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    11882655\n",
       "0     2096937\n",
       "Name: treatment, dtype: Int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.treatment.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fbe8b106b90>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAERCAYAAAB2CKBkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAPlklEQVR4nO3df6zdd13H8edrLYMYBiK9RNd2dGIZVhxMbwaBKOOHSUe0NThHKxOEQiUyjILEGcwgIybywyxANrHqmFvC6piI1RT7B0NH3Aa7C2OuXQpNQXYd0gsbv0SZJW//uOeys9Nze0+b+72np5/nI7nZ/X6/n3Pue0vT577ne873pqqQJLXrjHEPIEkaL0MgSY0zBJLUOEMgSY0zBJLUOEMgSY2byBAkuS7JkST3jbD26iT39L6+kOSbKzGjJE2KTOLnCJL8IvBd4IaqevYJPO7NwAVV9brOhpOkCTORZwRVdRvwUP++JM9I8s9J7k7y6STPGvLQ7cBNKzKkJE2I1eMeYBntAt5YVV9M8jzgWuAlCweTPB04F7h1TPNJ0inptAhBkicCLwA+mmRh9+MHlm0DbqmqH6zkbJJ0qjstQsD8S1zfrKrnHmfNNuBNKzSPJE2MibxGMKiqvg18KcmvA2TecxaOJzkPeApwx5hGlKRT1kSGIMlNzP+lfl6S2SQ7gFcBO5J8HtgPbO17yHZgd03iW6QkqWMT+fZRSdLymcgzAknS8pm4i8Vr1qypDRs2jHsMSZood99999eramrYsYkLwYYNG5iZmRn3GJI0UZL8x2LHfGlIkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkho3cZ8sXg4//7Ybxj2CTkF3v/fV4x5BGgvPCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhrXWQiSXJfkSJL7Fjn+qiT39r5uT/KcrmaRJC2uyzOC64HNxzn+JeBFVXU+8C5gV4ezSJIW0dmvqqyq25JsOM7x2/s27wTWdTWLJGlxp8o1gh3AJxY7mGRnkpkkM3Nzcys4liSd/sYegiQvZj4Ef7jYmqraVVXTVTU9NTW1csNJUgM6e2loFEnOB/4KuLiqvjHOWSSpVWM7I0hyDvAx4Der6gvjmkOSWtfZGUGSm4CLgDVJZoF3AI8DqKoPAVcCTwWuTQJwtKqmu5pHkjRcl+8a2r7E8dcDr+/q50uSRjP2i8WSpPEyBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUuM5CkOS6JEeS3LfI8ST5QJJDSe5N8nNdzSJJWlyXZwTXA5uPc/xiYGPvayfw5x3OIklaRGchqKrbgIeOs2QrcEPNuxP40SQ/0dU8kqThxnmNYC3wQN/2bG/fMZLsTDKTZGZubm5FhpOkVowzBBmyr4YtrKpdVTVdVdNTU1MdjyVJbRlnCGaB9X3b64AHxzSLJDVrnCHYA7y69+6h5wPfqqqvjnEeSWrS6q6eOMlNwEXAmiSzwDuAxwFU1YeAvcDLgUPA94DXdjWLJGlxnYWgqrYvcbyAN3X18yVJo/GTxZLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUOEMgSY3rNARJNic5mORQkiuGHD8nyaeSfC7JvUle3uU8kqRjdRaCJKuAa4CLgU3A9iSbBpb9MXBzVV0AbAOu7WoeSdJwXZ4RXAgcqqrDVfUIsBvYOrCmgCf1vn8y8GCH80iShugyBGuBB/q2Z3v7+r0TuCzJLLAXePOwJ0qyM8lMkpm5ubkuZpWkZnUZggzZVwPb24Hrq2od8HLgxiTHzFRVu6pquqqmp6amOhhVktrVZQhmgfV92+s49qWfHcDNAFV1B/AEYE2HM0mSBnQZgruAjUnOTXIm8xeD9wys+QrwUoAkP818CHztR5JWUGchqKqjwOXAPuB+5t8dtD/JVUm29Ja9FXhDks8DNwG/VVWDLx9Jkjq0epRFST5ZVS9dat+gqtrL/EXg/n1X9n1/AHjh6ONKkpbbcUOQ5AnAjwBrkjyFRy8APwk4u+PZJEkrYKkzgt8Gfo/5v/Tv5tEQfJv5D4tJkibccUNQVe8H3p/kzVX1wRWaSZK0gka6RlBVH0zyAmBD/2Oq6oaO5pIkrZBRLxbfCDwDuAf4QW93AYZAkibcSCEApoFNvrVTkk4/o36O4D7gx7scRJI0HqOeEawBDiT5LPD9hZ1VtWXxh0iSJsGoIXhnl0NIksZn1HcN/WvXg0iSxmPUdw19h0dvIX0m8Djgv6vqSYs/SpI0CUY9IzirfzvJrzL/G8gkSRPupO4+WlUfB16yzLNIksZg1JeGXtG3eQbznyvwMwWSdBoY9V1Dv9L3/VHgyxz7i+glSRNo1GsEr+16EEnSeIx0jSDJuiR/n+RIkq8l+bsk67oeTpLUvVEvFn+Y+d83fDawFvjH3j5J0oQbNQRTVfXhqjra+7oemOpwLknSChk1BF9PclmSVb2vy4BvdDmYJGlljBqC1wGXAv8FfBW4BPACsiSdBkZ9++i7gNdU1cMASX4MeB/zgZAkTbBRzwjOX4gAQFU9BFyw1IOSbE5yMMmhJFcssubSJAeS7E/ykRHnkSQtk1HPCM5I8pSBM4LjPjbJKuAa4JeAWeCuJHuq6kDfmo3AHwEvrKqHkzztZP4lJEknb9QQ/Blwe5JbmL+1xKXAnyzxmAuBQ1V1GCDJbuY/jXygb80bgGsWAlNVR05gdknSMhjppaGqugH4NeBrwBzwiqq6cYmHrQUe6Nue7e3r90zgmUn+LcmdSTaPNrYkabmMekZA7yWdA0sufFSGPc2Qn78RuAhYB3w6ybOr6puPeaJkJ7AT4JxzzjmBESRJSzmp21CPaBZY37e9DnhwyJp/qKr/q6ovAQeZD8NjVNWuqpququmpKT/HJknLqcsQ3AVsTHJukjOBbczfpqLfx4EXAyRZw/xLRYc7nEmSNKCzEFTVUeByYB9wP3BzVe1PclWSLb1l+4BvJDkAfAp4W1X5iWVJWkEjXyM4GVW1F9g7sO/Kvu8LeEvvS5I0Bl2+NCRJmgCGQJIaZwgkqXGGQJIaZwgkqXGGQJIaZwgkqXGGQJIaZwgkqXGGQJIaZwgkqXGGQJIaZwgkqXGGQJIaZwgkqXGGQJIaZwgkqXGGQJIaZwgkqXGGQJIaZwgkqXGGQJIa12kIkmxOcjDJoSRXHGfdJUkqyXSX80iSjtVZCJKsAq4BLgY2AduTbBqy7izgd4HPdDWLJGlxXZ4RXAgcqqrDVfUIsBvYOmTdu4D3AP/b4SySpEV0GYK1wAN927O9fT+U5AJgfVX9U4dzSJKOo8sQZMi++uHB5AzgauCtSz5RsjPJTJKZubm5ZRxRktRlCGaB9X3b64AH+7bPAp4N/EuSLwPPB/YMu2BcVbuqarqqpqempjocWZLa02UI7gI2Jjk3yZnANmDPwsGq+lZVramqDVW1AbgT2FJVMx3OJEka0FkIquoocDmwD7gfuLmq9ie5KsmWrn6uJOnErO7yyatqL7B3YN+Vi6y9qMtZJEnD+cliSWqcIZCkxhkCSWqcIZCkxhkCSWqcIZCkxhkCSWqcIZCkxhkCSWqcIZCkxhkCSWqcIZCkxhkCSWqcIZCkxhkCSWqcIZCkxhkCSWqcIZCkxhkCSWqcIZCkxhkCSWqcIZCkxhkCSWpcpyFIsjnJwSSHklwx5PhbkhxIcm+STyZ5epfzSJKO1VkIkqwCrgEuBjYB25NsGlj2OWC6qs4HbgHe09U8kqThujwjuBA4VFWHq+oRYDewtX9BVX2qqr7X27wTWNfhPJKkIboMwVrggb7t2d6+xewAPjHsQJKdSWaSzMzNzS3jiJKkLkOQIftq6MLkMmAaeO+w41W1q6qmq2p6ampqGUeUJK3u8LlngfV92+uABwcXJXkZ8HbgRVX1/Q7nkSQN0eUZwV3AxiTnJjkT2Abs6V+Q5ALgL4AtVXWkw1kkSYvoLARVdRS4HNgH3A/cXFX7k1yVZEtv2XuBJwIfTXJPkj2LPJ0kqSNdvjREVe0F9g7su7Lv+5d1+fMlSUvzk8WS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1LhOfx+BpBPzlat+dtwj6BR0zpX/3unze0YgSY0zBJLUOEMgSY0zBJLUOEMgSY0zBJLUuE5DkGRzkoNJDiW5Ysjxxyf5297xzyTZ0OU8kqRjdRaCJKuAa4CLgU3A9iSbBpbtAB6uqp8Crgbe3dU8kqThujwjuBA4VFWHq+oRYDewdWDNVuBvet/fArw0STqcSZI0oMtPFq8FHujbngWet9iaqjqa5FvAU4Gv9y9KshPY2dv8bpKDnUzcpjUM/PduVd73mnGPoMfyz+aCdyzL/x8/fbEDXYZg2OR1Emuoql3AruUYSo+VZKaqpsc9hzTIP5srp8uXhmaB9X3b64AHF1uTZDXwZOChDmeSJA3oMgR3ARuTnJvkTGAbsGdgzR5g4Xz8EuDWqjrmjECS1J3OXhrqveZ/ObAPWAVcV1X7k1wFzFTVHuCvgRuTHGL+TGBbV/NoUb7kplOVfzZXSPwfcElqm58slqTGGQJJapwhaNRSt/+QxiXJdUmOJLlv3LO0whA0aMTbf0jjcj2wedxDtMQQtGmU239IY1FVt+HniVaUIWjTsNt/rB3TLJLGzBC0aaRbe0hqgyFo0yi3/5DUCEPQplFu/yGpEYagQVV1FFi4/cf9wM1VtX+8U0nzktwE3AGcl2Q2yY5xz3S68xYTktQ4zwgkqXGGQJIaZwgkqXGGQJIaZwgkqXGGQDpJSc5OcssSa27v/XNDkt9YmcmkE+PbR6UVkOQi4A+q6pfHPYs0yDMCaQRJ3p3kd/q235nkrQv3zE/yM0k+m+SeJPcm2djb/93eQ/4U+IXe8d9f+X8DaXGGQBrNbuCVfduXMn+rjgVvBN5fVc8Fppm/n1O/K4BPV9Vzq+rqTieVTtDqcQ8gTYKq+lySpyU5G5gCHga+0rfkDuDtSdYBH6uqL45jTulkeEYgje4W4BLmzwx29x+oqo8AW4D/AfYlecnKjyedHM8IpNHtBv4SWAO8CHj8woEkPwkcrqoP9L4/H7i177HfAc5awVmlkXlGII2od4fWs4D/rKqvDhx+JXBfknuAZwE3DBy/Fzia5PNeLNapxrePSlLjPCOQpMYZAklqnCGQpMYZAklqnCGQpMYZAklqnCGQpMb9P21MpxGRf0sAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(x=dataset.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    13322663\n",
       "1      656929\n",
       "Name: visit, dtype: Int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.target.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>visit</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>treatment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.961799</td>\n",
       "      <td>0.038201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.951457</td>\n",
       "      <td>0.048543</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "visit             0         1\n",
       "treatment                    \n",
       "0          0.961799  0.038201\n",
       "1          0.951457  0.048543"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(dataset.treatment, dataset.target, normalize='index')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just note that the target and treatment groups are not balanced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optimizing the size of the dataset for low memory environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in dataset.data.columns:\n",
    "    dataset.data[c] = pd.to_numeric(dataset.data[c], downcast='float')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.treatment = dataset.treatment.astype('int8')\n",
    "dataset.target = dataset.target.astype('int8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a binary classification problem definition we stratify train set by splitting target `0/1` column. In uplift modeling we have two columns instead of one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train shape: (9785714, 12)\n",
      "Validation shape: (4193878, 12)\n"
     ]
    }
   ],
   "source": [
    "stratify_cols = pd.concat([dataset.treatment, dataset.target], axis=1)\n",
    "\n",
    "X_train, X_val, trmnt_train, trmnt_val, y_train, y_val = train_test_split(\n",
    "    dataset.data,\n",
    "    dataset.treatment,\n",
    "    dataset.target,\n",
    "    stratify=stratify_cols,\n",
    "    test_size=0.3,\n",
    "    random_state=31\n",
    ")\n",
    "\n",
    "print(f\"Train shape: {X_train.shape}\")\n",
    "print(f\"Validation shape: {X_val.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "treatment_model = lgb.LGBMClassifier(random_state=31)\n",
    "control_model = lgb.LGBMClassifier(random_state=31)\n",
    "tm = TwoModels(estimator_trmnt = treatment_model, estimator_ctrl = control_model, method='vanilla')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "tm = tm.fit(X_train, y_train, trmnt_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "uplift_tm = tm.predict(X_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Qini coefficient on full data: 0.0848\n"
     ]
    }
   ],
   "source": [
    "# AUQC = area under Qini curve = Qini coefficient\n",
    "auqc = qini_auc_score(y_val, uplift_tm, trmnt_val) \n",
    "print(f\"Qini coefficient on full data: {auqc:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgcAAAGSCAYAAAB34mJWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXxdZbn3/8+VpE3SNp1baJN0gKYJhdYClYLMVIEyVXE44EAFjigOiAd9REQBxZ96zgEFH1DhgUNBBFQUOQoqIoM4l0EoUmiBpE1bOjdpOie9fn/cayU76U6yM+zs1eT7fr32a+9973utde9Suq593ZO5OyIiIiKxvFw3QERERJJFwYGIiIi0ouBAREREWlFwICIiIq0oOBAREZFWFByIiIhIKwoOREREpBUFByLSJWZ2lZn9v96uKyLJYVoESURSmdlHgSuAg4F64OfAl9y9LpftEpG+o8yBiDQzsyuAbwNfAEYARwNTgN+Z2aAcNq1dZlaQ6zaI9DcKDkQEADMbDlwHfMbdf+Pue9y9GvgAMBX4YFTvWjP7UfR6ipm5mS00sxVmtsHMvpxyzua67VxzgZm9YGb1Zva6mZ0elVeb2TvTnSflmheb2QrgD2b2GzP7dJtz/9PMzo1eV5nZY2a2ycxeNbMP9Mofmkg/peBARGLvAIoI3QjN3L0BeBQ4tYNjjwMqgXnAV83skM4uZmZHAXcTshQjgROA6i6090TgEOA04MfA+SnnngFMBn5tZkOBx6I646N6t5rZoV24lsiAouBARGJjgQ3u3pjmszXAuA6Ovc7dd7j7P4F/Am/L4HoXA3e6+2PuvtfdV7n70i6091p33+buO4BfALPNbHL02YeAn7v7LuAsoNrd/8fdG939OeBB4H1duJbIgKLgQERiG4Cx7fThTwDWd3DsWymvtwPDMrheOfB65s3bx8r4hbtvBX4NnBcVnQfcG72eDMw1sy3xgxA8HNiDa4v0awoORCT2F2AXcG5qYZSWnw881cvXW0mYEZHONmBIyvt0N/K2U63uA843s2OAYuCJlOs85e4jUx7D3P3SHrRdpF9TcCAiAERTFa8Dvmdmp5vZIDObAvyUkFW4t4PDu+MO4EIzm2dmeWZWamZV0WcvAOdFbZhDZl0AjxCyBF8DHnD3vVH5r4DpZvaR6HyDzOztmYyLEBmoFByISDN3/0/gKuC/ga3Am4Rf8O909229fK2/AxcC3wHqCJmJeMzAVwhZhc2EgOXHGZxvF2Ew5TtT60ddDqcSuhpWE7pAvg0U9tJXEel3tAiSiLTLzC4i3JyPdfcVuW6PiPQNBQci0iEz+wiwx93vz3VbRKRvKDgQERGRVjTmQERERFpRcCCSEGZ2vJm92tt1RUS6SsGBSEK4+x/dvbK368q+zKzQzO6M9nR4y8z+o5P6n4vq1UXHFaZ8NtvM/hh9VmtmX23nHNdEe0Kk7hkx2sweiPak2GBm90Z7XIjklIIDEdkv9PLui9cCFYSpkycD/yfe9CnNdU8DriTsGzEFOIgwgyP2Y+BpYDRhv4dLzeycNuc4mLBWw5o2p78eGBWd82DggKhtIjml4ECkj5nZ4Wb2nJltjX413m9m15vZSWZWm1Kv2sw+b2YvRr9KHzCzouizVnU7uNaV0W6HW83sX2b2npTPWu2YmLLbYUH0frSZ/Y+ZrTazzWb2UCfXGmtmv4qWKN4U/ZrOiz4rN7Ofm9l6M9toZv83Ks8zs6vNrMbM1pnZ3WY2ok17mndfjMqPNrM/R9f5p5mdlPmffrMLgK+7+2Z3fwW4HfhoO3UXAne4+8vuvhn4epu6U4B73b3J3V8HngHabur0f4EvArvblE8FHnL3+mgRql+kOVakzyk4EOlDZjYYeAi4h/BL86fAezs45APA6YSbyCzav4G153XgeGAE4dfuj8xsQobH3kNYAOlQwm6G3+mk/hVALWGDpgMIiym5meUTVimsIdxIS4F4WuRHo8fJhF/Pwwg30lTNuy+aWSlhD4XrCX9+nwceNLNxAGZ2a+oeCm0eL0Z1RgETCRtExf5J+zflQ9PUPcDMxkTvvwtcEK28WAkcA/w+rmxm7wd2u/sjac59C3CWmY2K2vVewg6YIjml4ECkbx0NDAK+6+573P1nwD86qH+zu692903A/wKzu3Ixd/9pdPxed38AWAYc1dlxUQAxH/hE9Ot6j7t3trfCHsIGTZOj+n/0MFf6KMLN+AvRLoo73f2Z6JgPATe6+xvR1tBfIiybnNqFkLr74oeBR9z9keg7PQYsBs6Ivu8n2+yhkPqYFZ0v3hSqLuUadUBJO99rWJq6pNT/FaHLYAewlJBl+Ef05zgM+P+Ay9s593PAYGBj9GgCbm2nrkifUXAg0rcmAqu89QIjNR3U785uh83M7AIze8FadiM8jLA1c2fKgU1RGj1T/wUsB35nZm+Y2ZUp56ppZyvoibT+/jVAASHzEFuZ8noy8H5rvcPicYSgJFMN0XPqwL/hhOWi26vfti7AVjMbDfyGsJ9DEeG7nmZmn4zqXAfc4+5vtnPunwKvEQKN4YRMz4/aqSvSZxQciPStNUCpmVlK2aRsXMjMJhP60j8NjHH3kcASIL52RzsfrgRGm9nITK/n7lvd/Qp3Pwg4G/gPM5sXnWtSOwMKV9OynwKEP4tGYG3qqdu06542GYGh7v6t6Dv/wMwa2nm8HLVzM+G/w9tSzvs24OV2vtrLaequdfeNhK6QJne/290b3b2W0GVyRlR3HnCZhZkObxGCh5+Y2RdTzvXDKDPSAPwg5ViRnFFwINK3/kK4+V1mZgVmdi4ZpPm7aSjhxroewMwuJGQOYi8AJ5jZpGgQ4JfiD9x9DaHv+9aoP3yQmZ3Q0cXM7CwzmxYFPvWEFHkT8HfCzfhbZjbUzIrM7NjosPuAz5nZ1JQU/APtZBkg/Ko+28xOM7P86FwnmVlZ1O5PRNsxp3ukjim4G7g6+m5VwMeAu9q55t3AxWY2IxoXcHVK3dfCV7cPRoMrDwT+jZYxCvMIf+azo8dq4OOEsQYQupT+3cyKzawYuITW4xtEckLBgUgfcvfdwLmEQXibCTeSn2fpWv8CbiAEJGuBmcCfUj5/DHgAeBF4ltB3nuojhHEES4F1tN9vHqsgDMRriK55q7s/6e5NhEzCNGAFYdDiv0XH3EkY+Pg0YQfIncBnOvhOK4EFhMGO6wmZhC/Q9X/LriGk8GsIu0H+l7v/BiAKlhrMbFJ0zd8A/wk8EdWviY7H3esJ/z0/R/jv+QIhO/ON6PON7v5W/CAES5ujLAHARYRBmrXAKkIm4qNd/C4ivU57K4jkmJndBdS6+9W5bouICChzICIiIm0oOBDZj6WkwNM9en2go5ld1c61NDdfpB9Rt4KIiIi0osyBiIiItKLgQERERFrpzV3O9mtjx471KVOm5LoZIiIifeLZZ5/d4O7j0n2m4CAyZcoUFi9enOtmiIiI9Akza3fpdnUriIiISCsKDkRERKQVBQciIiLSioIDERERaUXBgYiIiLSi4EBERERayVpwYGZ3mtk6M1uSUvaAmb0QParN7IWofIqZ7Uj57AcpxxxpZi+Z2XIzuznaKx4zG21mj5nZsuh5VFRuUb3lZvaimR2Rre8oIiLSH2Uzc3AXcHpqgbv/m7vPdvfZwIO03sf+9fgzd/9ESvn3gUsIe8VXpJzzSuBxd68AHo/eA8xPqXtJdLyIiIhkKGvBgbs/DWxK91n06/8DwH0dncPMJgDD3f0vHnaIuht4d/TxAmBR9HpRm/K7PfgrMDI6j4iIiGQgV2MOjgfWuvuylLKpZva8mT1lZsdHZaVAbUqd2qgM4AB3XwMQPY9POWZlO8e0YmaXmNliM1u8fv36nn0jERGRfiJXwcH5tM4arAEmufvhwH8APzaz4YClObazPaYzPsbdb3P3Oe4+Z9y4tMtLi4iIDDh9vreCmRUA5wJHxmXuvgvYFb1+1sxeB6YTfvWXpRxeBqyOXq81swnuvibqNlgXldcC5e0cs//YtAn+9rdct0JERHLtHe+AESP69JK52HjpncBSd2/uLjCzccAmd28ys4MIgwnfcPdNZrbVzI4G/gZcAHwvOuxhYCHwrej5lynlnzaz+4G5QF3c/bBf+T//B+64I9etEBGRXLvsMrjppj69ZNaCAzO7DzgJGGtmtcA17n4HcB77DkQ8AfiamTUCTcAn3D0ezHgpYeZDMfBo9IAQFPzEzC4GVgDvj8ofAc4AlgPbgQt7/cv1hbVrobISFi3qvK6IiPRPCxbA1q19ftmsBQfufn475R9NU/YgYWpjuvqLgcPSlG8E5qUpd+BTXWxu8tTVwYQJMHdurlsiIiK5UlwMTU19flmtkJhU9fUwfHiuWyEiIrmUl6fgQFLU1fX5ABQREUmY/HzYu7fPL6vgIKmUORAREWUOpJm7MgciIqLMgaTYsSNEisociIgMbMocSLO6uvCszIGIyMCmzIE0q68Pz8ociIgMbMocSLM4OFDmQERkYFPmQJrF3QrKHIiIDGzKHEgzdSuIiAgocyApNCBRREQgBAfKHAigzIGIiATqVpBmGnMgIiKgbgVJUV8PQ4ZAQdY2zRQRkf2BMgfSTEsni4gIKHMgKbTpkoiIgDIHkkKZAxERAWUOJIUyByIiAsocSIr6emUOREREmQNJUVenzIGIiGgRJEmhbgUREQF1K0hk717YulXdCiIiom4FiTQ0gLsyByIiosyBRLTpkoiIxJQ5EECbLomISAtlDgRQ5kBERFoocyCAMgciItJCmQMBlDkQEZEWyhwIoMyBiIi0UOZAAGUORESkhTIHAoTMgRkMHZrrloiISK5p+WQBQnBQUhJSSSIiMrCpW0GA0K2gLgUREQF1K0hEmy6JiEisv2UOzOxOM1tnZktSyq41s1Vm9kL0OCPlsy+Z2XIze9XMTkspPz0qW25mV6aUTzWzv5nZMjN7wMwGR+WF0fvl0edTsvUds0KZAxERifXDzMFdwOlpyr/j7rOjxyMAZjYDOA84NDrmVjPLN7N84BZgPjADOD+qC/Dt6FwVwGbg4qj8YmCzu08DvhPV238ocyAiIrH+ljlw96eBTRlWXwDc7+673P1NYDlwVPRY7u5vuPtu4H5ggZkZcArws+j4RcC7U861KHr9M2BeVH//oMyBiIjE+mHmoD2fNrMXo26HUVFZKbAypU5tVNZe+Rhgi7s3tilvda7o87qo/j7M7BIzW2xmi9evX9/zb9YblDkQEZFYPHOtjwOEvg4Ovg8cDMwG1gA3ROXpftl7N8o7Ote+he63ufscd58zbty4jtrdd5Q5EBGRWH5+eO7PwYG7r3X3JnffC9xO6DaA8Mu/PKVqGbC6g/INwEgzK2hT3upc0ecjyLx7I7f27IEdO5Q5EBGRIM4c9PG4gz4NDsxsQsrb9wDxTIaHgfOimQZTgQrg78A/gIpoZsJgwqDFh93dgSeA90XHLwR+mXKuhdHr9wF/iOonX7yvgjIHIiICLZmDPg4OCjqv0j1mdh9wEjDWzGqBa4CTzGw2Ic1fDXwcwN1fNrOfAP8CGoFPuXtTdJ5PA78F8oE73f3l6BJfBO43s+uB54E7ovI7gHvMbDkhY3Betr5jr9OmSyIikipH3QpZCw7c/fw0xXekKYvrfwP4RpryR4BH0pS/QUu3RGr5TuD9XWpsUsSbLik4EBERGBjdCtIJdSuIiEiqgTAgUTqhbgUREUmlzIE0dysocyAiIqDMgaDMgYiItKbMgShzICIirShzINTXQ0EBFBXluiUiIpIEyhxI89LJ+9E+USIikkU5WgRJwUGSaNMlERFJpW4F0aZLIiLSiroVRJkDERFpRZkDoa5OwYGIiLRQ5kCor1e3goiItFDmQNStICIirShzMMC5a0CiiIi0pszBALdrF+zZo8yBiIi0UOZggNPSySIi0pYyBwOcNl0SEZG2lDkY4JQ5EBGRtrR88gCnzIGIiLSlboUBTpkDERFpS90KA5wyByIi0pYyBwOcMgciItKWMgcDXJw5KCnJbTtERCQ5lDkY4OrqoKgIBg/OdUtERCQpkpo5MLPpZva4mS2J3s8ys6uz37QBRpsuiYhIWwnOHNwOfAnYA+DuLwLnZbNRA5I2XRIRkbaSmjkAhrj739uUNWajMQOaNl0SEZG2ErwI0gYzOxhwADN7H7Amq60aiJQ5EBGRtnLUrVCQQZ1PAbcBVWa2CngT+HBWWzUQ1dVBRUWuWyEiIkmSo26FToMDd38DeKeZDQXy3H1r9ps1AClzICIibSU1c2BmI4ELgClAgZkB4O6XZbVlA43GHIiISFtJzRwAjwB/BV4C+jZ0GSjclTkQEZF9JTVzABS5+39kvSUDWUNDCBCUORARkVQJnsp4j5l9zMwmmNno+NHZQWZ2p5mtixdPisr+y8yWmtmLZvaLqMsCM5tiZjvM7IXo8YOUY440s5fMbLmZ3WxRv0bUjsfMbFn0PCoqt6je8ug6R3T5T6WvadMlERFJJ8GLIO0G/gv4C/Bs9FicwXF3Aae3KXsMOMzdZwGvERZXir3u7rOjxydSyr8PXAJURI/4nFcCj7t7BfB49B5gfkrdS6Ljky3edEnBgYiIpEpw5uA/gGnuPsXdp0aPgzo7yN2fBja1Kfudu8cLKP0VKOvoHGY2ARju7n9xdwfuBt4dfbwAWBS9XtSm/G4P/gqMjM6TXHHmQN0KIiKSKsGZg5eB7Vm49kXAoynvp5rZ82b2lJkdH5WVArUpdWqjMoAD3H0NQPQ8PuWYle0ck0zqVhARkXQSPFuhCXjBzJ4AdsWFPZnKaGZfJizBfG9UtAaY5O4bzexI4CEzOxSwNId7Z6fP9Bgzu4TQ9cCkSZMyaXp2xN0KyhyIiEiqHC2fnElw8FD06BVmthA4C5gXdRXg7ruIAg93f9bMXgemE371p3Y9lAGro9drzWyCu6+Jug3WReW1QHk7x7Ti7rcRVn9kzpw5nQUd2aPMgYiIpJPUbgV3XwTcR8tgxB9HZV1mZqcDXwTOcfftKeXjzCw/en0QYTDhG1F3wVYzOzqapXAB8MvosIeBhdHrhW3KL4hmLRwN1MXdD4mlzIGIiKST1G4FMzuJMOCvmpCyLzezhdGAw46Ouw84CRhrZrXANYTZCYXAY9GMxL9GMxNOAL5mZo2EboxPuHs8mPFSwsyHYsIYhXicwreAn5jZxcAK4P1R+SPAGcBywliJCzv7jjkXZw6GDcttO0REJFkSvAjSDcCp7v4qgJlNJ2QSjuzoIHc/P03xHe3UfRB4sJ3PFgOHpSnfCMxLU+6EzaL2H3V1UFLS8pdAREQEEj2VcVAcGAC4+2vAoOw1aQDS0skiIpJOgjMHi83sDuCe6P2HCGMPpLdo0yUREUknqWMOCH3+nwIuI4w5eBq4NZuNGnCUORARkXTi4CCBmYMC4CZ3vxEgmlVQmNVWDTR1dTByZK5bISIiSZSXl8gxB48TZgrEioHfZ6c5A1R9vboVREQkvfz8RAYHRe7eEL+JXg/JXpMGoLo6dSuIiEh6+fnJWwQJ2Ja67XG0vPGO7DVpAFLmQERE2pODboVMxhxcDvzUzOIliCcA52WvSQNMUxNs26bMgYiIpJeDzEEmwcGLQBVQSZitsJTMMg6SCW3XLCIiHUnogMS/uPsed1/i7i+5+x7gL9lu2IChTZdERKQjScocmNmBQClQbGaH07IV8nA0ILH3aNMlERHpSMLGHJwGfJSw5fGNKeVbgauy2KaBRZkDERHpSJIyB9G2zIvM7L3RxkiSDcociIhIRxKWOYgdZmaHti10969loT0DjzIHIiLSkSRlDlI0pLwuAs4CXslOcwagOHOg4EBERNJJYubA3W9IfW9m/w08nLUWDTSayigiIh1J6PLJbQ0BDurthgxYdXXhP/wQTQAREZE0ktitYGYvAR69zQfGARpv0Fvi7ZrNOq8rIiIDTxK7FQhjDGKNwFp3b8xSewaeODgQERFJJ4kbL7l7DTASOBt4DzAj240aUOrqNN5ARETal8Tlk83ss8C9wPjoca+ZfSbbDRswlDkQEZGOJHHMAXAxMNfdtwGY2bcJeyt8L5sNGzDq6mDChFy3QkREkiqJmQPCngqprWqiZZ8F6SllDkREpCMJzRz8D/A3M/tF9P7dwB3Za9IAozEHIiLSkSTOVnD3G83sSeA4QsbgQnd/PtsNGzCUORARkY7kYBGkTDIHuPtzwHNZbsvAs2tXeCg4EBGR9iRxKqNkkZZOFhGRziR0QKJkizZdEhGRzihzMMAocyAiIp1J0oBEM9tKy54KrT4C3N31c7enlDkQEZHOJGkqo7uX9GVDBiRlDkREpDNJyhy0ZWbjgaL4vbuvyEqLBpI4OFDmQERE2pPEMQdmdo6ZLQPeBJ4CqoFHs9yugSHuVlDmQERE2pPQ2QpfB44GXnP3qcA84E9ZbdVAocyBiIh0JomZA2CPu28E8swsz92fAGZncnIzu9PM1pnZkpSy0Wb2mJkti55HReVmZjeb2XIze9HMjkg5ZmFUf5mZLUwpP9LMXoqOudnMrKNrJE5dHRQWhoeIiEg6Cc0cbDGzYcDThO2abwIaMzz/XcDpbcquBB539wrg8eg9wHygInpcAnwfwo0euAaYCxwFXJNys/9+VDc+7vROrpEsWjpZREQ6k4PlkzMJDhYA24HPAb8BXgfOyuTk7v40sCnN+RZFrxcRNnKKy+/24K/ASDObAJwGPObum9x9M/AYcHr02XB3/4u7O3B3m3Olu0ayaNMlERHpTEK7Fb7q7nvdvdHdF7n7zcAXe3DNA9x9DUD0PD4qLwVWptSrjco6Kq9NU97RNZJFmQMREelMQrsV3pWmbH5vN4SwuFJb3o3yzC9odomZLTazxevXr+/Kob2jrk7BgYiIdCxJmQMzu9TMXgIqowGC8eNN4MUeXHNt1CVA9LwuKq8FylPqlQGrOykvS1Pe0TVacffb3H2Ou88ZN25cD75SN9XXq1tBREQ6lrDMwY+Bs4GHo+f4caS7f7gH13wYiGccLAR+mVJ+QTRr4WigLuoS+C1wqpmNigYingr8Nvpsq5kdHc1SuKDNudJdI1mUORARkc4kbPnkOqAOON/M8oEDovrDzGxYJiskmtl9wEnAWDOrJcw6+BbwEzO7GFgBvD+q/ghwBrCcMADywqgdm8zs68A/onpfc/d4kOOlhBkRxYSFmeLFmdq7RrIocyAiIp1J4vLJZvZp4FpgLRCHLg7M6uxYdz+/nY/mpanrwKfaOc+dwJ1pyhcDh6Up35juGonirgGJIiLSuSRlDlJcDlRGN1zpLdu3h0hQmQMREelIwsYcxFYSuhekN2npZBERyUQOFkHKJHPwBvCkmf0a2BUXuvuNWWvVQKBNl0REJBMJ7VZYET0GRw/pDcociIhIJpI4INHdrwMws6Huvi37TRoglDkQEZFMJGkRpJiZHWNm/wJeid6/zcxuzXrL+jtlDkREJBMJHZD4XcLmRxsB3P2fwAnZbNSAEGcOFByIiEhHkpg5AHD3lW2K+jaE6Y/izIG6FUREpCNJHHMArDSzdwBuZoOBy4i6GKQH4sxBSUlu2yEiIsmW0MzBJwgrF8ZbJM+mnZUMpQvq62HoUCjIJD4TEZEBKy8vrKrrXdp4uEcyma2wAfhQH7RlYNGmSyIikon8/PC8d2/L6yzLZG+FqcBngCmp9d39nOw1awDQpksiIpKJvCjJ39SUnOAAeAi4A/hfWjZekp7SpksiIpKJOCDow0GJmQQHO9395qy3ZKCpq1PmQEREOpfardBHMgkObjKza4Df0Xpvheey1qqBoL4eSktz3QoREUm61G6FPpJJcDAT+AhwCi3dCh69l+5S5kBERDKR0MzBe4CD3H13thszoGjMgYiIZCIHmYNM1jn4JzAy2w0ZUJqaYOtWBQciItK5hGYODgCWmtk/aD3mQFMZu6uhITyrW0FERDqT0DEH12S9FQONNl0SEZFMJTFz4O5PmdlkoMLdf29mQ4C+WYWhv9KmSyIikqkkjjkws48BPwN+GBWVEhZGku5S5kBERDKVg0WQMhmQ+CngWKAewN2XAeOz2ah+T5kDERHJVA66FTIJDnalTmM0swLCOgfSXcociIhIppLYrQA8ZWZXAcVm9i7gp4R9FqS7lDkQEZFMJTRzcCWwHngJ+DjwCHB1NhvV78XBgTIHIiLSmaRNZTSzfGCRu38YuL1vmjQA1NWBGQwbluuWiIhI0iUtc+DuTcA4MxvcR+0ZGOKlk81y3RIREUm6pGUOItXAn8zsYWBbXOjuN2arUf1eXZ26FEREJDNJXAQJWB098oCS7DZngKiv12BEERHJTBIzB+5+XV80ZEBR5kBERDKVtDEHAGb2mJmNTHk/ysx+m91m9XPKHIiISKYSus7BOHffEr9x981ohcSeUeZAREQyldDlk5vMbFL8JtqESSsk9oQyByIikqkkdisAXwaeMbN7zOwe4GngS929oJlVmtkLKY96M7vczK41s1Up5WekHPMlM1tuZq+a2Wkp5adHZcvN7MqU8qlm9jczW2ZmDyRuKqYyByIikqkkdiu4+2+AI4AHgJ8AR7p7t8ccuPur7j7b3WcDRwLbgV9EH38n/szdHwEwsxnAecChwOnArWaWHy3QdAswH5gBnB/VBfh2dK4KYDNwcXfb2+t274adO5U5EBGRzCQ0cwBQCGwC6oAZZnZCL11/HvC6u9d0UGcBcL+773L3N4HlwFHRY7m7vxFtDHU/sMDMDDiFsM00wCLg3b3U3p7bujU8K3MgIiKZSOJURjP7NvBvwMtAHLY4oXuhp84D7kt5/2kzuwBYDFwRDX4sBf6aUqc2KgNY2aZ8LjAG2OLujWnqt2JmlwCXAEyaNCldld4X78iozIGIiGQioZmDdwOV7n6mu58dPc7p6YWjcQDnEHZ5BPg+cDAwG1gD3BBXTXO4d6N830L329x9jrvPGTduXBda3wPadElERLoiiWMOgDeAQVm49nzgOXdfC+Dua929yd33EjZ5OiqqVwuUpxxXRlixsb3yDcBIMytoU54MceZAwYGIiGQiocsnbwdeMLPHgV1xobtf1sNrn09Kl4KZTXD3NdHb9wBLotcPAz82s9ir/ZIAACAASURBVBuBiUAF8HdChqDCzKYCqwhdFB90dzezJ4D3EcYhLAR+2cO29p44c6BuBRERyUQSxxwQbs4P9+ZFzWwI8C7g4ynF/2lmswldANXxZ+7+spn9BPgX0Ah8KtotEjP7NPBbIB+4091fjs71ReB+M7seeB64ozfb3yPKHIiISFfkYBGkTPZWWNTbF3X37YSBg6llH+mg/jeAb6QpfwR4JE35G7R0SySLMgciItIVSepWMLOfuPsHzOwl0gzoc/dZWW1Zf6XMgYiIdEXCuhU+Gz2f1RcNGTDq62HQICgqynVLRERkf5CkzEE8OLCTBYqkq+Klky3djEsREZE2EjqVUXqTNl0SEZGuSOgiSNKbtOmSiIh0RZIyB9G6BvHyydJblDkQEZGuSNKYA2CCmZ0InGNm99NmWWJ3fy6rLeuv6uuhvLzzeiIiIpC42QpfBa4kLD98Y5vPnLDzoXRVXR0cdliuWyEiIvuLJC2C5O4/A35mZl9x96/3WYv6u/p6jTkQEZHMxZmDhHQrAODuXzezc4AToqIn3f1X2W1WP+WuAYkiItI1OcgcdDpbwcy+SVgQ6V/R47NRmXTVzp3Q2KgBiSIikrmEDUiMnQnMjrZSxswWETYz+lI2G9YvaelkERHpqiRNZWxjZMpr/eztLm26JCIiXZXQzME3gefN7AnCdMYTUNage5Q5EBGRrkrYVEYA3P0+M3sSeDshOPiiu7+V7Yb1S8ociIhIVyU0cxBvwvRwltvS/ylzICIiXZXgMQfSG5Q5EBGRrsrBOgcKDvpSHBwocyAiIpkyC4+kZA7MLM/MlvRVY/o9dSuIiEh35OcnJziI1jb4p5lN6qP29G/19VBcDIMG5bolIiKyP8nPT9yAxAnAy2b2d2BbXOju52StVf2Vlk4WEZHuyMtL1lRG4Lqst2KgqK/XYEQREem6pGUO3P0pM5sMVLj7781sCJCf/ab1Q8ociIhId/Rx5iCTjZc+BvwM+GFUVAo8lM1G9VvKHIiISHf0ceYgk6mMnwKOBeoB3H0ZMD6bjeq3lDkQEZHuSFrmANjl7rvjN2ZWAHj2mtSPKXMgIiLdkcDMwVNmdhVQbGbvAn4K/G92m9VPKXMgIiLdkcDMwZXAeuAl4OPAI8DV2WxUv7R3L2zdqsyBiIh0XR8vgpTJbIW9ZrYI+BuhO+FVd1e3Qlc1NIC7MgciItJ1eXnJmspoZmcCPwBeJ2zZPNXMPu7uj2a7cf2KNl0SEZHuSlrmALgBONndlwOY2cHArwEFB12hTZdERKS7EjggcV0cGETeANZlqT39lzZdEhGR7krK8slmdm708mUzewT4CWHMwfuBf/RB2/oXdSuIiEh3JWj55LNTXq8FToxerwdGZa1F/ZUyByIi0l1JyRy4+4XZvLCZVQNbgSag0d3nmNlo4AFgClANfMDdN5uZATcBZwDbgY+6+3PReRbSMrXyendfFJUfCdwFFBOmX342p7MslDkQEZHuStqYAzObamY3mtnPzezh+NFL1z/Z3We7+5zo/ZXA4+5eATwevQeYD1REj0uA70dtGw1cA8wFjgKuMbM4q/H9qG583Om91ObuUeZARES6KymZgxQPAXcQVkXMdtiyADgper0IeBL4YlR+d/TL/69mNtLMJkR1H3P3TQBm9hhwupk9CQx3979E5XcD7yaXMyzizEFJSc6aICIi+6kEjTmI7XT3m7NwbQd+Z2YO/NDdbwMOcPc1AO6+xsziDZ5KgZUpx9ZGZR2V16Ypb8XMLiFkF5g0aVJvfKf21dWFwCAvkwkiIiIiKRKYObjJzK4BfgfsigvjPv8eONbdV0cBwGNmtrSDupamzLtR3rogBCS3AcyZMye74xG06ZKIiHRXAhdBmgl8BDiFlm4Fj953m7uvjp7XmdkvCGMG1prZhChrMIGW9RRqgfKUw8uA1VH5SW3Kn4zKy9LUzx1tuiQiIt2VtAGJwHuAg9z9RHc/OXr0KDAws6FmVhK/Bk4FlgAPAwujaguBX0avHwYusOBooC7qfvgtcKqZjYoGIp4K/Db6bKuZHR3NdLgg5Vy5ocyBiIh0VwK7Ff4JjKR3V0U8APhFuG9TAPzY3X9jZv8AfmJmFwMrCAsuQZiKeAawnDCV8UIAd99kZl+nZVGmr8WDE4FLaZnK+Ci5Xu65rg5Gj85pE0REZD+VwAGJBwBLoxt36piDc7p7UXd/A3hbmvKNwLw05Q58qp1z3QncmaZ8MXBYd9vY6+rrYcqUXLdCRET2R3l5sGdPn10uk+Dgmqy3YiBQt4KIiHRXfj7s2tV5vV7SaXDg7k/1RUP6PQ1IFBGR7kramAMz20rLNMDBwCBgm7vrTpepxkbYvl2ZAxER6Z6kjTlw91ZL+pnZuwnTDiVT8eqIyhyIiEh39HHmoMvL9bn7Q/RwjYMBR5suiYhITyRtESQzOzflbR4whzSrDUoHtOmSiIj0RF5esroVgLNTXjcStlJekJXW9FfKHIiISE8kLXPg7hf2RUP6NWUORESkJ5IyINHMvtrBce7uX89Ce/onZQ5ERKQnEjSVcVuasqHAxcAYQMFBppQ5EBGRnkhK5sDdb4hfR5skfZawp8H9wA3tHSdpaCqjiIj0RIIyB5jZaOA/gA8Bi4Aj3H1zXzSsX6mvD1HfkCG5bomIiOyPkpI5MLP/As4FbgNmuntDn7Wqv4mXTg67UIqIiHRNghZBugKYCFwNrDaz+uix1czq+6Z5/YQ2XRIRkZ5ISubA3bu8eqK0Q5suiYhITyQocyC9RZkDERHppoYGWLcxH0/KgETpJXV1MHFirlshIiI5tmsXrFwJL74In/88vPlmKC8oCBv4BhuApa0eN/EMH2YrQ3ZCUVH226ngoC/U18Mhh+S6FSIi+wX3cKOMH3v2hOe9e8NnvfmAlvM3NYVrmLWuk+66qWW7d4fj8/Jg507YuhW2bIHHH4dt22DUKHjtNVi6tOWaYTeCN4lv/o2NcSDwKrAx5U+jCJjOHziMOvKY99xe3vGO7Cf9FRz0BY05EJFOxDfE3bvDzSp+jl/HN6O9e1tuYvEj9f2uXS1j1xobw81q507YsaPldXy+pqaWR3yd+LFnTzhXfN1UZi030PjGGj/v2gVr1sCBB+77WdvXu3eH+o2Nrc/Vh9nzPlDHhAmvUly8lOLipYwcuZQtW15lx45luO9prjVy5AGUlVVx4IHvo7KyitLSKoYNq2Lu3Enk5+cxdixMntx3rVZw0Bc05kAkkRobwy+77dvDI93rtmXxzTX1Jp7J6127wj8FBQUtN+f45h/fNHMpPz88Bg0Kj7w8GDy4JRBIFf9iNgvfJz42Lw82bIADDoD161vK8/OhuLjldXyMWes6eXnhefDgcJ34/Hl5Le1IbYtZWD4mtWznzpZ6Q4aE93FbAYYObX2uhoZwzfif6G3bWs4dH5P63LYdIUjay+bNK6mvX8rmzUvZtGkpGze+ytq1S9myZQ1r1sR/xgXk509j9uwqDjroHCorq5g5s4rDDqvkgANGUlQUvm8SJKQZ/Vj8L4kyBzIApf5y3b493CDjX7Fx/+reveH99u3hBtLQAIsXQ2VlSM02NIS6tbXhf6Pdu8M5du1qOd+uXeEcTU0tN67GxnDOOB1tFl5v3drSpu7ckOObZ0FBeI5vqPENL/V1XCf+R3/iRFiyBI48svVNMd0NsqAgHBdnAfLyWh6DB4fvWlDQ8t3MQnlDQ7jmmDHhuw4e3NLm4cPDn0FBAZSXw+bNod5bb4W2vfVWuHnGN8hRo0LdpqZwox08ONz843/O4pXhx4yBwsLw556X1/pm3J+Wd9m5czs1Na9RXb2U6uql1NS82vy8a9eO5nrDho2kvPwQ5s49nYqKKg45JAQBlZVTKSkZRFFR8v9cFBxkmzZdkv2Ae7iR1NW1POrrW15v3RpuGG+9BWvXtqSod+9uSXXHv5S3bw+fbdiQnV/DJSWtb3iDB4cb2ODBrX9h5uWFuvEvVGi5sRYWtpyjsDDc+IYNC8FJUVH4fnG9YcPC8WVlLd+5qCi8njgRVq+GSZNazt/YGM7T9tdlEm8G48aF5zFjwvOoUZ0fU1ra8rqkJH2dJH7XTLk7Gze+1RwApAYBa9bUNNczMw44YCrl5VUsWHAKlZVVHHpoJTNnVlFWNo7iYmPQoBx+kR5ScJBt2nRJ+oB7uClv2RIemzfv+zpd2QsvhLh127bUkdLtGz4cRo8ON874phwbNCjcFMeMCeXDh4f38a/n2loYPz4cH6et45t4/Ks+/oVdXx9uvEOGhBtQQUFLoJGX13otmIkTQ1l8rb6muH//tGfPblauXL5PAFBdvZRt21rW+SsqGkpZWRWHHHIcZ5/97xxySOgGmDGjgpEjiygqCn//+hsFB9mmTZekm3bvhnXrwk08/hW/aRNs3Ag33BBuSiNHwvLloX+3s5t7UVH4FTx0aHh++eVQfuyxLZ8VFYVfy8XF4cZcVBReFxeHYCA/PxxTWRlSzfvzLyMZGLZs2Zg2AFi9+g2aUkY+jh1bRnl5FaeeegHTp7dkAaZOLaWoyCgszOGXyAEFB9mmboUBb+/ekI6vrw83+pdfhpqaMGgrHuy2ZUtI18dp+9WrW5JOHTnsMKiqgqOPbn0zHzIkPIYObXkd/6oeO7alL7kn+uOvJdk/NTY2smZN9T4BQHX1UrZs2dBcb9CgQkpLpzNlymzmzTuPqqpKDjusikMPnc64cSXNYzxEwUH2qVuh39q2LUzZWrkSHnsspMuXLQu/4jduhKefzvxcxcXhhj1yZDjPQQeFvzIlJeHGXljY0v89bFhLX+/o0SHIEBkIGhrqqal5dZ8AYOXKZezZs7u53siR4ykrq+LYY8+loqKSGTOqmDWrimnTJjN0aD6Fhfv3uIi+oOAg25Q52O80NoZf7suXw+9/D088AX/9a7gJb98ebsxbtoTXbaX2yUNIv5eVhRHkc+eGG/y2bTBhQggG4oFx8a/wsrL2B3mJDAR79+5l3bra6Mb/KjU1LUHA+vWrm+vl5eUzceI0ysurOProM6mqquKww8J4gAkTRidqWuD+SH902abMQc65w+uvh8F39fVh2dKXX4ZDDw0p/JUrww36pZdCH3o8J7mtvLzwi37kyPCfM+7zj0fEl5WFNH7soIMYcP2UIpnauXMHK1a8tk8WoKbmVXbubIm8hw4dQXn5Icye/a7maYGzZlVRWXkQw4cPprBQXVzZoOAg2zQgsc/s3QtvvAE/+EGYw710KbzyCvzjH+l/5f/pT+Hmvn17mNI1blz4zzR3bstI+VGjwuv4F0hVldKRIpkK0wLXtgoAamqW8uabS3nrrRo8WnoxTAucQllZJWeffWI0LTCsDVBePn6/nxa4P1JwkG11dS0dxtIrtm2DP/8ZLrkEpkwJ/fVLl8KqVWGEf6ykJMw/f/vbw0C9Qw4Jv/aHDg3PQ4a0Pu/06RqMJNIdjY17qK19vVUGIH40NLSMrC0sHEJZWSVVVcdw5pkXNi8ONGNGBaNGFffbaYH7IwUH2aalk7utvh5+9rPQ//+Vr4SyYcNC/32sujrMnS8pgfnzWwYFnndeyALEv/K175VIz9XVbdqnG6C6eimrVr3ealrgmDETKSurYt68DzF9elXzgMCpU0sZMiSveVlmSS4FB9mmTZcy1tAAf/gDLFjQfp38fDj+eDjuuNB1EA/eKy8PgYOI9ExTUxNr1tS06gaIX2/evL65XkHBYEpLK5g0aSYnn/z+lAGB0xk3brimBe7nFBxkmzIH+9i4Ee66K0z/++1vO647bx6cdVbICOTlhdX3xo/vk2aK9GvbtzekzQKsXLmM3bt3NdcbPnws5eVVHHPMAioqWrIAFRVTNC2wH1NwkG3KHLB8OXz84yEr0JF3vjMM/jvuuDCOQF0BIj3j7qxbt6rNbIAwIHD9+lXN9fLy8pkw4SDKyqo46qj5rXYLnDBhDEVFWg1zoOnz4MDMyoG7gQOBvcBt7n6TmV0LfAyI81ZXufsj0TFfAi4GmoDL3P23UfnpwE1APvD/3P1bUflU4H5gNPAc8BF3Txmq1ofq68OctgFi92744hfDznNLloQV/9IZOhSuv74lI6BAQKT7du3aycqVy/bJAtTUvMqOHdua6w0ZMpzy8ipmz57HtGkt0wKrqg7WtEBpJReZg0bgCnd/zsxKgGfN7LHos++4+3+nVjazGcB5wKHAROD3ZjY9+vgW4F1ALfAPM3vY3f8FfDs61/1m9gNCYPH9rH+zdAZAt8L27WH635IlHde74gqYPTu8VjAg0jXuzubN69POCFizprp5WiDA+PGTKS+v4owzjk+ZFljJpEkHUlxszYt0ibSnz4MDd18DrIlebzWzV4DSDg5ZANzv7ruAN81sOXBU9Nlyd38DwMzuBxZE5zsF+GBUZxFwLbkKDvpxt8J73ws//3n6z6ZPhy99KawPoLUBRDIXpgW+sc+AwJqaV6mv39xcr7CwmNLSSqZPn8sZZyxM2S1wOqNHD6GwUAMCpftyOubAzKYAhwN/A44FPm1mFwCLCdmFzYTA4a8ph9XSEkysbFM+FxgDbHH3xjT1217/EuASgEnxhuy9yb3fZQ5WrIDJk9N/9slPwjHHKCsgkon6+s1pVwdcuXI5TU0tW2yOHj2B8vIqTjrpvGhaYNgt8OCDyzUtULImZ8GBmQ0DHgQud/d6M/s+8HXAo+cbgIuAdH/tHUjXM+Yd1N+30P024DaAOXPmpK3TI9u2hWX79tPMgXuYUXDaae3XmT8fPvhBBQQi6TQ1NfHWWyv2CQCqq5eyadPa5noFBYOYOLGC8vIZHH/8uVRWVjYPCBw/foT2CZA+l5O/bmY2iBAY3OvuPwdw97Upn98O/Cp6WwuUpxxeBsS7b6Qr3wCMNLOCKHuQWr9v7YebLjU1df6PUGEh3HabBhKKxLZvb2DFitf2CQBqal5j9+6dzfVKSkZTXn4Ic+eeRUVFZfOAwOnTpzJsWIGmBUpi5GK2ggF3AK+4+40p5ROi8QgA7wHi4W0PAz82sxsJAxIrgL8TMgQV0cyEVYRBix90dzezJ4D3EWYsLAR+mf1vlsZ+tulSR/8oXX112GEQFBDIwOTurF+/ep8AoLp6KWvXtvRw5uXlceCBB1FeXsURR7yLysqwONDMmVWUlo7VtEDZL+Qic3As8BHgJTN7ISq7CjjfzGYTugCqgY8DuPvLZvYT4F+EmQ6fcvcmADP7NPBbwlTGO9395eh8XwTuN7PrgecJwUjfS3jmwL39aUsf/WhYgEiBgAw0YVrg8lZBQDwocPv2lrW7i4tLKC+v4rDDTuQ976lq3ifgkEOmMXx4ofYJkP1aLmYrPEP6cQGPdHDMN4BvpCl/JN1x0QyGo9qW97mEZg4uuQRuvz39Z3fcEbYvVlAg/Zm7s2XLhrRZgNWr32Tv3r3NdcePn0RZWRWnn34h06eHaYGzZlUxefIETQvMkT179lBbW8vOnTs7rywUFRVRVlbGoC6krDTEJZsSljl45pmwL0FbN90UFiMCBQXSvzQ2NrJq1Rv7BADV1Uupr9/UXG/w4CJKS6dz8MFHcuqpH6KqKt4tcDpjxgzVPgEJU1tbS0lJCVOmTME0SKNDYdvsjdTW1jJ16tSMj1NwkE0JyBy013Vw3XVh4caKCo2Clv1fQ0NddNN/tdVGQStXLqexcU9zvVGjDqSsrJITT3x/826BM2dWMW3aJE0L3I/s3LlTgUGGzIwxY8awfv36ziun0G0hm+LMQQ6Cg927w6yCdO65RxkC2f/s3bu3eVpg2yzAxo0t63Tn5xcwceI0ysurOPbYBa32CTjggJGaFthPKDDIXHf+rPS/SDblKHOQ7u/BJZeELgUFBZJ0O3Zso6bmtX0CgBUrXmPXrh3N9YYNG0V5eRVvf/t8pk1L3S1wKiUlgygqUhZAssfM+PCHP8w999wDhC6sCRMmMHfuXH71q191cnSLKVOmsHjxYsaOHdujOr1NwUE21deHHYb6oLPyqafgpJP2Lb/mGpg2TUGBJIu7s2HDmrRbBr/11ormembGAQdMpby8isMPn7fPtMDiYtO0QMmJoUOHsmTJEnbs2EFxcTGPPfYYpaUd7QSwf1FwkE19tHRyul9HP/whDBmioEBya/fuXdTWvp52s6Dt27c21ysqGkpZWRUzZhzPOee0nhY4cmSRpgVKIs2fP59f//rXvO997+O+++7j/PPP549//CMAmzZt4qKLLuKNN95gyJAh3HbbbcyaNYuNGzdy/vnns379eo466qhWG2b96Ec/4uabb2b37t3MnTuXW2+9lfwcjYRVcJBNWdx0aceOcPNva8oU+PrXFRRI39qyZeM+GwVVVy9l1ao3Wk0LHDu2jPLyKk47bWHztMCZMyuZOrVU0wKlWy6/HF54ofN6XTF7Nnz3u53XO++88/ja177GWWedxYsvvshFF13UHBxcc801HH744Tz00EP84Q9/4IILLuCFF17guuuu47jjjuOrX/0qv/71r7ntttsAeOWVV3jggQf405/+xKBBg/jkJz/JvffeywUXXNC7Xy5DCg6yKUuZg3SZgh/8AA48EMrKev1yIkDoU12zpjptFqCubmNzvUGDCiktnc6UKYczb9750W6BVRx66HTGjh2maYHSb8yaNYvq6mruu+8+zjjjjFafPfPMMzz44IMAnHLKKWzcuJG6ujqefvppfh5tZ3vmmWcyatQoAB5//HGeffZZ3v72twOwY8cOxo8f34ffpjUFB9nUy5mDG26Az3++ddk3vxkCAmUKpLc0NNSnHQuwcuWyVtMCR44cT1lZFccf/14qKqo45JBKZs2qYtq0yQwdmq99AqRPZPILP5vOOeccPv/5z/Pkk0+ycWNLkJzaXRCLZw2kmz3g7ixcuJBvfvOb2WtsFyg4yKb6eigv77xeBtr+XbrmGpg5M3QjiHTV3r17WbeuNm0WYMOGNc318vLym6cFvuMdZ1NZWclhh4VpgRMmjNa0QBnwLrroIkaMGMHMmTN58sknm8tPOOEE7r33Xr7yla/w5JNPMnbsWIYPH95cfvXVV/Poo4+yefNmAObNm8eCBQv43Oc+x/jx49m0aRNbt25l8uTJOfle+t86m3ohc/CNb4RNj2JVVfDlLytTIJnZuXM7K1Ys2ycAWLHiNXbu3N5cb+jQEZSXH8IRR5xGRUUVVVUhC1BZeRDDhw+msFADAkXSKSsr47Of/ew+5ddeey0XXnghs2bNYsiQISxatAgIYxHOP/98jjjiCE488UQmTZoEwIwZM7j++us59dRT2bt3L4MGDeKWW27JWXBg6VIfA9GcOXN88eLFvXvSkhL42Mfgxhs7r5vG/Pnwm9+0vL/9djj88F5qm/QbYXnUtWkGBL7KW2/VNKc3w7TAKZSXVzFlSiWVlfGAwCrKy8drWqDsN1555RUO0S+kLkn3Z2Zmz7r7nHT1lTnIlqYmaGjoduagbTeCVjWUPXt2p50WWFPzKg0Ndc31CguHUF5eRVXVOzjzzAubpwXOmFHBqFHFmhYoIp1ScJAtW6M53F0MDtoGBW97WxiEqMBg4Kir25Q2AFi16nWampqa640ZU0p5eRXz5n2oeZ+AWbOqmDq1VPsEiEiPKDjIlnjp5C5MZWz7D/m118KZZ+of+P6oqalpn2mB8QyBzZtbNkgpKBhMaWkFkybN5OST309VVVU0IHA648YN17RAEckKBQfZ0sVNl1IDgCOPDAt7KFuw/9u2bWuraYHx6xUrXmPPnt3N9UaMGEdZWSXHHLOAiorUfQKmaFqgiPQ5BQfZEgcHGWQOUv/RnzcPPvpRBQb7E3dn7draVgFATc1S3nxzKevXr2qul5eXz4QJB1FWVsXcufOZPr1lt8AJE8ZQVIQGBIpIIig4yJYMd2RMDQy+852wc6Ik086dO1i5ctk+WYCamlfZsWNbc70hQ4ZTXl7F7NnzmDYt7BMwa1YVVVUHa1qgiOwXFBxkSwaZg9TA4LrrFBgkgbuzadO6fQKA6uqlrFlT3WrVs/HjJ1NeXsUZZxzfPC1w1qwqyssP0D4BIv1cfn4+M2fOpLGxkalTp3LPPfcwcuTIHp+3urqas846iyVLlvRCK7tPwUG2dJI5SA0MZs8OaxpI32ls3JMyLfDVVpsFbd26pbleYWExpaWVTJ8+lzPOWJiyW2AFo0cPobBQAwJFBqLi4mJeiHZ8WrhwIbfccgtf/vKXc9yq3qPgIFs6yBykBgb//u9w4olKM2dLff3mfTIA1dVLqa19naamxuZ6o0dPoLy8ipNPPr95WuDMmZUcfHC5pgWKSIeOOeYYXnzxRQAaGhpYsGABmzdvZs+ePVx//fUsWLCA6upq5s+fz3HHHcef//xnSktL+eUvf0lxcTHPPvssF110EUOGDOG4445rPu/OnTu59NJLWbx4MQUFBdx4442cfPLJ3HXXXTz00EM0NTWxZMkSrrjiCnbv3s0999xDYWEhjzzyCKNHj+7Rd1JwkC11deGOP3Roq+LUG8x3vwspfw+km8K0wJo2swLCgMDNm9c11ysoGMTEiRWUlx/KiSe+l8rKkAU49NDpjB8/QvsEiOyHLr/88uZf8L1l9uzZfDfDHZ2ampp4/PHHufjiiwEoKiriF7/4BcOHD2fDhg0cffTRnHPOOQAsW7aM++67j9tvv50PfOADPPjgg3z4wx/mwgsv5Hvf+x4nnngiX/jCF5rPfcsttwDw0ksvsXTpUk499VRee+01AJYsWcLzzz/Pzp07mTZtGt/+9rd5/vnn+dznPsfdd9/N5Zdf3qM/A/1TmC319aFLISUaiHbmBOALX4CjjspBu/Zj27c3RDMBWmcBVqx4jd27dzXXKykZTXn5IRx99NnN0wJnzqxk+vSpDBtWoGmBItJjO3bsYPbs2VRXV3PkkUfyrne9Cwjjlq666iqefvpp8vLyWLVqFWvXrgVg6tSpzJ49YvHkoAAAEOFJREFUG4AjjzyS6upq6urq2LJlCyeeeCIAH/nIR3j00UeBsO3zZz7zGQCqqqqYPHlyc3Bw8sknU1JSQklJCSNGjODss88GYObMmc1ZjJ5QcJAtbTZdWrECtkRd2VddBeeem6N2JZy7s27dqrRbBq9bV9tcLy8vjwMPPIjy8irmzDktGhBYycyZVZSWjtW0QJEBItNf+L0tHnNQV1fHWWedxS233MJll13Gvffey/r163n22WcZNGgQU6ZMYefOnQAUFhY2H5+fn8+OHTtw97RbOEP6bZ9jqefKy8trfp+Xl0djY2N7h2VMwUG21Ne3Gm+QurHWKafkoD0Js2vXTlauXL7PZkE1Na+yfXtDc73i4hLKy6uYNevk5t0Cw4DAaYwYUahpgSKSUyNGjODmm29mwYIFXHrppdTV1TF+/HgGDRrEE088QU1NTYfHjxw5khEjRvDMM89w3HHHce+99zZ/Fm/vfMopp/Daa6+xYsUKKisree6557L9tRQcZE1K5iDqNgLg7ruhF2a77BfcnS1bNuyTAaiuXsrq1W+2mRY4ibKyKubPv4iKiv+/vfsPjqLM8zj+/hICEwm7/Igokig/kxBDIIogeyLW8fMowPPIKkoVlHonSK27HmVtybHlnYqnt7h7HnV6wnmgltmFKFccZckGWRVZihPOVRAXIai4giiSrCH4Aww890c/GXriAJnshGEmn1dV13Q//XQ/neepmnyn+3n6KYoOC7zsst4aFigi57Xy8nKGDh3KypUrmTlzJlOnTmX48OEMGzaM4uLisx6/YsWKaIfEiRMnRtPnzZvH3LlzGTJkCB07duTpp5+OuWPQljRls5f0KZuHD4deveCll6LPt3NyYNOm5BVxvmhs/JYDBz5sNk9A8HnkyJ+i+Tp1ipCfX0RBQTH9+xeFZgsspGfPLponQERaRFM2J05TNp8v6uth0CCef/5U0rJlqbucZGho+CJuX4CPP94bMyywe/eLKSgoZsyYG0PDAosZOPBSDQsUEUkDCg7aih+tcOONwebs2ekxX8LJkyf59NM/xn0UUFf3WTRfVlZHPyywmGuu+evosMDS0iIuuqibhgWKiKQxfX23lSNHOJF7qkPiuHEpvJY4vv76Sz76aE/MI4BgfQ/Hj38TzZeb252CgmJGjJjMoEGn5gkoLOxH167ZGhYoIpKBFBy0hePH4Ztv+MdfnhrKmIq7Bs45Dh8++J07AB99tJtPP/1jNF+HDh3o1asvBQXFlJePo6iomNLS4uiwwJwc07BAEZF2RMFBW/CvTq4nuHOwYkXbFnf8+LGYYYHh1wV/9VVDNF9OTi59+hRRUjKaadOKQ/MEDKRbtwiRiIYFioiIgoO24SddOkJw52DIkOSctvmwwKYA4MCBDzh58mQ0X15ePgUFxUycOJvCwlOzBfbte4mGBYqIyFkpOGgLoTsHTzyR2KGNjY188knzYYFBEFBfXxvNl53dmT59Cunbt5yxY29m8ODgUcDllxeSl5erYYEiIm0oPGXz4MGDeeaZZ7jgggtafPymTZuYO3cu2dnZbNmyhZycnBYfu2bNGgoLCykpKWnNpbdIxgYHZjYJ+DcgC3jKOffIOSs8dOega9f4WY4erWffvt2hDoG7/bDAGhobv43m69atF/n5xYwePT06T0BZWREDBlxGly5Z6hAoIpIC4SmbZ86cyZNPPsn8+fNbdOyJEyeorKzknnvu4dZbb0247DVr1jBlyhQFB4kysyzgcWA8sB/YZmZrnXN/OBflH/v8CJ2BerrSrdtHbNkSewdg3773OHz4YDR/hw5ZXHLJQAoKivnBD6ZSXBzcBSgtLeLii7trWKCIyHls9OjR0cmOnnvuOZYsWcLx48cZOXIkTzzxBFlZWeTm5jJ//nyqq6uZOnUqVVVVVFdXs2HDBiorK1m8eDFVVVUcO3aMG264gfvvvx+AZ599lkcffRQzo6ysjDvvvJO1a9eyceNGFi1axOrVqxkwYEDS/6ZM/ZczAtjrnPsAwMxWAtcD5yQ4WPyzV/gZcNRGM3XqqdkCu3T5PgUFg7nyyokMHNjUIbCIoqL+fO97nYhEdBdARCQhd98NSZ6ymWHDoIUTOjU2NrJu3TomTZrErl27WLVqFZs3byY7O5t58+ZRWVnJrFmz+PLLLyktLeWBBx4AYO/evUyZMoWKigrWr19PTU0NW7duxTnHtGnTeP311+nZsycPPfQQmzdvJi8vj7q6Onr06MG0adOix7aVTA0O+gAfh7b3AyObZzKzO4A7AC699NKkFX5x6YW8vKcnV4z7ITeOLI/OFlhQ0EvDAkVEMkDTlM0Q3Dm4/fbbWbZsGW+++SZXXXVVNE+vXr2AoI/C9OnT455r/fr1rF+/nvLycgCOHj1KTU0N27dvp6Kigry8PAB69OjR1n9WVKYGB/F+f39nEgnn3DJgGQRzKySr8L9dvRBYyPhknVBEROJL8ZTNYc45Zs+ezcMPP/yd/JFIhKzT9BJ3zrFgwQLmzJkTk75kyZLTTufc1jJ1VPt+oCC0nQ98kqJrERGRdmDs2LG88MILHDp0CIC6urqzTtkMMHHiRJYvX87Ro8F09QcOHODQoUOMHTuWqqoqamtro+cD6Nq1Kw0NDac9XzJkanCwDRhkZv3MrBMwA1ib4msSEZEMVlJSwqJFi5gwYQJlZWWMHz+egwcPnvW4CRMmcMsttzBq1CiGDBlCRUUFDQ0NXH755SxcuJAxY8YwdOjQ6GiIGTNmsHjxYsrLy3n//ffb5G/J2CmbzWwy8BjBUMblzrmHzpQ/6VM2i4hIm9CUzYnTlM2ec+4l4KVUX4eIiEi6ydTHCiIiItJKCg5EREQkhoIDERFJO5naX64ttKauFByIiEhaiUQi1NbWKkBoAecctbW1RCKRhI7L2A6JIiKSmfLz89m/fz+ff/55qi8lLUQiEfLz8xM6RsGBiIiklezsbPr165fqy8hoeqwgIiIiMRQciIiISAwFByIiIhIjY1+fnCgz+xw4+wwZLZcHHE7i+SRxaoPUUxukntogtc7n+r/MOXdhvB0KDtqImf3f6d5ZLeeG2iD11AappzZIrXStfz1WEBERkRgKDkRERCSGgoO2syzVFyBqg/OA2iD11AaplZb1rz4HIiIiEkN3DkRERCSGgoM2YGaTzGy3me01s3tTfT3pwMyWm9khM9sZSuthZi+bWY3/7O7TzcyW+PrdYWZXhI6Z7fPXmNnsUPqVZvaOP2aJmVlry8hUZlZgZq+a2S4ze9fMfuLT1Q7niJlFzGyrmW33bXC/T+9nZm/4+lllZp18eme/vdfv7xs61wKfvtvMJobS434/taaMTGVmWWb2lpm96LfbX/0757QkcQGygPeB/kAnYDtQkurrOt8X4FrgCmBnKO3nwL1+/V7gX/z6ZGAdYMDVwBs+vQfwgf/s7te7+31bgVH+mHXAX7WmjExegN7AFX69K7AHKFE7nNM2MCDXr2cDb/i/uwqY4dOfBO706/OAJ/36DGCVXy/x3z2dgX7+OynrTN9PiZaRyQswH/gV8GJr6iYT6j/ljZBpi//iqw5tLwAWpPq60mEB+hIbHOwGevv13sBuv74UuLl5PuBmYGkofalP6w28F0qP5ku0jFTX0Tluj/8BxqsdUlb/FwC/B0YSvESno0+PfscA1cAov97R57Pm3ztN+U73/eSPSaiMVNdPG9Z7PvBb4C+BF1tTN5lQ/3qskHx9gI9D2/t9miTuIufcQQD/2cunn66Oz5S+P056a8poF/yty3KCX65qh3PI39J+GzgEvEzwS/ML51yjzxKug2j9+P31QE8Sb5uerSgjUz0G/BQ46bdbUzdpX/8KDpLP4qRpSEhyna6OE01vTRkZz8xygdXA3c65I2fKGidN7fBncs6dcM4NI/gFOwIYHC+b/0xWG5ypnttNG5jZFOCQc+7NcHKcrBlf/woOkm8/UBDazgc+SdG1pLvPzKw3gP885NNPV8dnSs+Pk96aMjKamWUTBAaVzrn/9slqhxRwzn0BvEbQ56CbmXX0u8J1EK0fv//7QB2Jt83hVpSRif4CmGZm+4CVBI8WHqMd1r+Cg+TbBgzyPU87EXQgWZvia0pXa4Gmnu6zCZ6BN6XP8j3Zrwbq/a3oamCCmXX3vd0nEDy3Owg0mNnVvnf8rGbnSqSMjOXr5r+AXc65X4Z2qR3OETO70My6+fUcYBywC3gVqPDZmtdPU71VAK+44OH0WmCG7+neDxhE0Bk07veTPybRMjKOc26Bcy7fOdeXoG5ecc7NpD3Wf6o7f2TiQtDDeg/Bs8KFqb6edFiAXwMHgW8JIuXbCZ6r/Rao8Z89fF4DHvf1+w4wPHSe24C9frk1lD4c2OmP+XdOvQAs4TIydQGuIbhduQN42y+T1Q7ntA3KgLd8G+wE7vPp/Qn+uewFngc6+/SI397r9/cPnWuhr7fd+FEhPj3u91NrysjkBbiOU6MV2l396w2JIiIiEkOPFURERCSGggMRERGJoeBAREREYig4EBERkRgKDkRERCSGggORDGVmzsx+Edq+x8z+KUnnftrMKs6es8Xn+4dkness5Qwzs8mtOO41MxveFtckcj5ScCCSuY4Bf2Nmeam+kDAzy4qTnHBwcJrznM0wgnHmInIGCg5EMlcjsAz4++Y7mv/yN7Oj/vM6M9toZlVmtsfMHjGzmWa21czeMbMBodOMM7NNPt8Uf3yWmS02s21mtsPM5oTO+6qZ/YrgZUbha3kEyDGzt82s0qetMbM3zexdM7sjfJ1m9oCZvQGMMrPJZvaemf3OzJaY2Ys+XxczW+6v4y0zu96/ke4B4CZf1k3x8vnjc8xspf8bVgE5f3ZriKSRjmfPIiJp7HFgh5n9PIFjhhJM9lMHfAA85ZwbYWY/Ae4C7vb5+gJjgAHAq2Y2kOCVyPXOuavMrDOw2czW+/wjgFLn3Ifhwpxz95rZj1ww2VCT25xzdf4VwtvMbLVzrhboQjCt931mFiF4o+K1zrkPzezXoeMXErxm9jb/OuKtwAbgPoK3LP4IwMz+uXk+M9sAzAG+cs6VmVkZwdTJIu2G7hyIZDAXzKr4LPDjBA7b5pw76Jw7RvCK16Z/7u8QBARNqpxzJ51zNQRBRDHBPAqzLJhy+A2C1yIP8vm3Ng8MzuDHZrYd+F+CSWeaznGCYGIofHkfhM4ZDg4mAPf663iN4BW0l8Yp53T5rgWeA3DO7SB4nbFIu6E7ByKZ7zGCX74rQmmN+B8HfhKkTqF9x0LrJ0PbJ4n9zmj+7vWmqWfvcs5Vh3eY2XXAly25WJ93HDDKOfeVmb1G8E8b4Bvn3ImmrGc6DTDdObe72blHtjBf098j0i7pzoFIhnPO1QFVBJNZNdkHXOnXrweyW3HqH5pZB98PoT/BBDPVwJ0WTP2MmRWaWZcWnOvbpmMIpqT9kw8MigmmLI7nPaC/mfX12zeF9lUDd/nABzMr9+kNQNcW5HsdmOnTSgkmRBJpNxQciLQPvwDCoxb+ExhjZluBkbTwV30zu4GNwDpgrnPuG+Ap4A/A781sJ7CUlt2hXEbQN6IS+A3Q0cx2AA8SPFr4Dufc18A84Ddm9jvgM6De736QIODZ4a/jQZ/+KlDS1CHxDPn+A8j11/BTgj4LIu2GZmUUkbRlZrnOuaP+l//jQI1z7l9TfV0i6U53DkQknf2d70z4LsHjiKUpvh6RjKA7ByIiIhJDdw5EREQkhoIDERERiaHgQERERGIoOBAREZEYCg5EREQkhoIDERERifH/NlSnQPn6hFQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# with ideal Qini curve (red line)\n",
    "# perfect=True\n",
    "\n",
    "plot_qini_curve(y_val, uplift_tm, trmnt_val, perfect=True);"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
